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ABSTRACT 

The paper presents preliminary empirical and analytically derived 
information pertinent to three questions: To what extent does the 
format of a report affect the degree to which it conveys intended 
information? What range of qualitative reactions to a particular 
format might be expected from the intended audiences of a report? 
What are the computer programming costs and complexities associated 
with a given report format? 
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EXPLORATORY EXAMINATION OF ALTERNATIVE FORMATS FOR 
COMMUNICATING INSTRUCTIONAL PROFICIENCY STATUS INFORMATION 

Instructional proficiency status information may be cotmnunicated 
in several alternative formats. For exaii5)le, bar graphs, data tables, 
and natural language text provide alternative means of coninunicating 
such information. While cost, time, and scale of operation constrain 
the feasible bounds of format diversification, it should be possible to 
provide a reasonable number of options consistent with user preference. 
This paper reports the results of an exploratory study addressing the 
following questions: 

(a) To what extent does the format of a report affect the degree 
to which it conveys intended information? 

(b) What range of qualitative reactions to a particular format 
might be expected from the intended audiences of a report? 

(c) What are the computer programming costs and complexities 
associated 'with a given report format? 

Procedures 

Data on the first question was obtained by administering a test 
to measure the acquisition of equivalent information from a represen- 
tative report presented in each of three alternative formats. The tests 
were administered individually to 24 staff members of the Laboratory. 
The 24 subjects were randomly divided into six groups of four and 
presented with the three tests in different sequences, thus effecting 
a Latin Square design and reducing the impact of ordering on performance. 
Standardized instructions were read to each subject at the beginning of 
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the testing session, and the length of time necessary to complete each 
of the three tests was recorded. Sample tests and format alternatives 
are shown in Attachments 1 and 2. 

The second question was probed in conjunction with the testing by 
having subjects complete an open-ended evaluative questionnaire shown 
in Attachment 3. 

To obtain estimates pertinent to the third question copies of the 
format alternatives were referred to the Computer Center with a request 
for comments regarding cost-complexity. 

Results 

Tables 1 and 2 present performance results of the subjects on the 
tests measuring acquisition of equivalent information from alternate 
formats. The data reveal that the mean performance score on all tests 
was approximately 18 out of a possible score of 20. The speed with 
which subjects acquired information from the test formats appears to 
vary much more than the accuracy of data interpretation. 

Table 3 shows the format preference of subjects as indicated by 
their responses to the open-ended questionnaire. Twenty-one (88%) of 
the subjects selected data tables over the graph and text formats. 
Only one other person indicated a different preference by choosing the 
graph. Data on the frequency of positive and negative comments about 
each format are found in Attachment 4. 

Estimates made by computer staff on relative computer programming 
difficulties associated with the three formats are presented in Attachment 5, 
Table 1 of the attachment lists by headings (HDGS) the data and total 
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TABLE 1 

ACCURACY OF INTERPRETATION OF ALTERNATIVE FORMATS 



TABLE 


N 


_ SCORES 

X S.D. 


MEAN 


DIFFERENCES 


Graph (A) 


24 


18.16 


2.13 


AB 




.71 


Data (B) 














Tables 


24 


18.87 


.99 


BC 




.00 


Text (C) 


24 


18.87 


1.22 


AC 




.71 



TABLE 2 

TIME REQUIRED TO COMPLETE ALTERNATIVE FORMATS 



TABLE 


N 


TIME 
X S.D. 


MEAN 


DIFFERENCES 


Graph (A) 


24 


6:38 


1:36 


AB 




1:89* 


Data (B) 














Tables 


24 


4:49 


1:16 


BC 




:86* 


Text (C) 


24 


5:35 


1:57 


AC 




:21 



N = 24, df = 23 

* indicates P < .05 



tabu: 3 

format preferences as indicated on questionnaire 



FORMAT 


N 


% 


Graph (A) 


I 


(4) 


Data (B) 
Tables 


21 


(88) 


Text (C) 


0 


(0) 


No response 


2 


(8) 



number of print lines necessary for 10 units of reports. Table 2 shows 
the length of data print lines for each unit of each report format. In 
each case Format B (data tables) requires the least amount of computer 
processing. 

Summary and Discussion 

The purpose of this inquiry was to probe three considerations in 
making a selection among IMS Function 1 alternative formats. 

To what extent does the format of a report affect the degree 
to which it conveys intended information? 

The results indicate that the information conveyed was easily 
acquired by the subjects. In fact, it is likely that the high perfor- 
mance of the subjects would have been even higher had the poorer test 
items been improved or deleted. Also, the report format had virtually 
no effect on accurate acquisition of information. However, graphs and 
text required significantly more time for subjects to gather this 
information. 
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The subjects of this study are not presently teaching. However, 
many of them have taught before and can relate to the teacher's concerns 
about reports and their formats. The educational range of the subjects 
varied from high school through the Ph.D. level, but performance did 
not vary. Thid suggests that the performance of teachers (most of whom 
have college degrees and graduate credits) may not differ significantly 
from that of the subjects tested. 

What range of qualitative reactions to a particular format 

might be expected from the intended audiences of a report? 
The questionnaire results indicate that the range of qualitative ' ^ 
reactions to the three formats is very narrow. The data tables were 
almost unanimously preferred, and very few negative reactions were 
recorded. In addition, very few positive reactions to the graph and 
text formats were recorded, and the subjects concurred in their negative 
reactions. For example, 13 of the 22 subjects reacting to the graph 
format found it difficult to identify the exact percentage of students 
reaching criterion. Sixteen of them described the two page format as 
clumsy and awkward. 

What are the computer programming costs and complexities 

associated with a given report format? 
With respect to CPU time and possible print charges, there are 
probably no significant differences between the three formats when 
distributed in small volume. With a large volume of reports, the data 
table format would be less expensive to process. With regard to the 
time necessary to transmit data to the schools, data tables would take 
significantly less time to process in both small and large volume. 



ATTACHMENT 1 



REPORT #4 
QUESTIONS ON CONTENT 



1. Which instructional program is this report for? 

2. Which class is this report for? 

3. On which unit was no test given? 

4. Did the class take a test on Unit 2? 

5. What is the name of the skill associated with 
Outcome II? 

6. How many pages are in this report? 

7. What percentage of the students in this class 
reached criterion on Outcome I for Unit 1? 

8. What percentage of the students in this class 
reached criterion on Outcome IV for Unit 6? 

9. What percentage of the students in this class 
reached criterion on Outcome II for Unit 5? 

10. On which units did less than 807. of the class 
reach criterion for Outcome III? 

11. How many students did not reach criterion on 
Outcome I of Unit 2? 

12. What percentage of the students did not 
reach criterion on Outcome III of Unit 7? 

13. What percentage of the students did not 
reach criterion on Outcome IV of Unit 5? 

14. On Word Attack for Unit 3, percentage 

of the students reached criterion. 

1?. For Unit 3, on which Outcome did the class 
have the highest performance ? 

On which Outcome did the class have the lowest 
performance? 

16. For Unit 6, on which Outcome did the class 
perform best? 

On which Outcome did the class perform least 
well? 
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17. On Outcome IV, did the class show «ny improvement 
in perforaance between Unit 5 and Unit 6? 

18. Did the class do better on the Word Attack Skill 
of Unit 5 or Unit 6? 



5/14/71 
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ATTACHMENT 2 



Display 4(A). Bar Graph showing the percentage of pupils within a class 

reaching criterion (807. correct on an outcome) on cummulative 
Criterion Exercises. (Reported in sets of five.) 



REPORT #4 (A) 



PROGRAM: SYCSP 
UNITS: 1-5 



DISTRICT: L.A. (A) 



SCHOOL: 
TEACHER: 



FIRST ST. 
HANKS 



MAR. 31, 1971 
CLASS: 1 
PAGE 1 



PERCENT OF STUDENTS 
ACHIEVING CRITERION 
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OUTCOME 1 (WORDS) 



OUTCOME II (WORD ELEMENTS) 



PERCENT OF STUDENTS 
ACHIEVING CRITERION 
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OUTCOME III (WORD ATTACK) 
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OUTCOME IV (LETTER NAMES) 
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Display 4(A) . Bar Gtaph ihowirig the percentage of pupils within a class 

reaching criterion (80% correct on an outcome) on cunroulative 
Criterion Exercises. (Reported in sets of five.) 



REPORT #4(A) 



PROGRAM: SYCSP 
UNITS: 6-7 



DISTRICT; LOS ANGELES (A) 
SCHOOL; FIRST STREET 
TEACHER: HANKS 



MAY 15, 1971 
CLASS : 1 
PAGE 2 



PERCENT OF STUDENTS 
ACHIEVING CRITERION 
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OUTCOME I (WORDS) 



OUTCOME II (WORD ELEMENTS) 
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Display 4(B) . N:<meric data Indicating the percentage of pupils within 
a class reaching criterion (80% correct on an outcome) 
on cumroulative Criterion Exercises. 



REPORT #4(B) 

PROGRAM: FYCSP DISTRICT: L.A. (A) MAY 31, 1971 

UNITS: 1-7 SCHOOL: FIRST ST. CLASS: 2 

TEACHER: HANKS 



OUTCOME 



UNIT 

3 4 5 6 7 8 9 10 



I (WORDS) 

NO. OF STUDENTS 18 20 16 16 16 16 

7, OF CLASS 907. 1007o 807. 807. 807. 807. 

II (WORD ELEMENTS) 

NO. OF STUDENTS 20 16 12 4 4 4 

7. OF CLASS 1007. 807. 607. 207. 207. 207. 

Ill (WORD ATTACK) 

NO. OF STUDENTS 12 4 4 8 2 20 

7. OF CLASS 607. 207. 207. 407. 107. 1007. 

IV (LETTER NAMES) 

NO. OF STUDENTS 12 12 8 4 20 16 

7. OF CLASS 607. 607. 407, 207. 1007. 807. 
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Display 4(C). Statements describing the percentage of pupils within a 
class achieving masttry (807. correct on an outcome) on 
curamulative Criterion Exercises. 

REPORT #4(C) 

PROGRAM: FYCSP DISTRICT: L-A. (A) MAY 31, 1971 

UNITS: 1-7 SCHOOL: FIRST ST. CLASS: 4 

TEACHER: HANKS 

This report presents the number and percent of students within this class 
reaching criterion on the IMS Criterion Exercises given to date. 



Performance of this class on Outcome I (Words) 

16 students or 80% of the class reached criterion on Unit 1 
4 students or 20% of the class reached criterion on Unit 2 

16 students cr 80% of the class reached criterion on Unit 3 

NO TEST 

16 students or 807o of the class reached criterion on Unit 5 

12 students or 60% of the class reached criterion on Unit 6 

16 students or 80% of the class reached criterion on Unit 7 

Performance ot this class on Outcome II (Word Elements) 

20 students or 1007, of the class reached criterion on Unit 1 
16 students or 807. of the class reached criterion on Unit 2 
12 students or 60% of the class reached criterion on Unit 3 

NO TEST 

20 students or 1007. of the class reached criterion on Unit 5 
4 students or 207o of the class reached criterion on Unit 6 
4 students or 207. of the class reached criterion on Unit 7 

Performance of this class on Outcome III (Word Attack) 

20 students or 1007. of the class reached criterion on Unit 1 
4 students or 207. of the class reached criterion on Unit 2 
12 students or 607. of the class reached criterion on Unit 3 

NO TEST 

8 students or 40% of the class reached criterion on Unit 5 
20 students or 1007. of the class reached criterion on Unit 6 
16 students or 807» of the class reached criterion on Unit 7 

Performance of this class on Outcome IV (Letter Names) 

12 students or 607. of the class reached criterion on Unit 1 
12 students or 607. of the class reached criterion on Unit 2 
8 students or 407. of the class reached criterion on Unit 3 

NO TEST 

20 students or 1007. of the class reached criterion on Unit 5 
16 students or 807. of the class reached criterion on Unit 6 
16 students or 807» of the class reached criterion on Unit 7 
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ATTACHMENT 3 




With reference to Report #4, what factors do you like and what factors 
do you dislike about each Format, A, B> and C? 

A. like - 



dislike 



B. like - 




dislike - 



C. like - 



dislike 



Which Format do you prefer overall? 
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ATTACHMENT 4 
Frequency of Comaents Recorded on Questionnaire 



FORMAT 


NUMBER OF 
RESF(»<SES 


POSITIVE NUMBER OF 
C(»IMENTS RESPONSES 


NEGATIVE 
COMMENTS 


A 


2 


Quick comparisons are easy 


16 


2 pages are awkward , 
sloWy or clumsy 




2 


Allows for easy comparison 
of outcome performance across 
units 


13 


Difficult to locate 
percentages 




1 


Included "NO TEST" statement 


7 


Doesn't include number 
of students 




1 


Easy to locate area containing 
information 


3 


Asterisks are usually 
unpleasing 


1 


JL 


Increases or decreases in 
performance are clearly seen 


1 


Labels don't stand out 




1 


Prefers graphs^ 


1 


Difficult to compare 
units 


B 


14 


Compact') simple 9 easy to read 


1 
I 


Too much reading 




9 


Easy comparisons 


1 


Needs a "NO TEST" 
statement 




7 


All information present 








1 


All information is on one page 








1 


It is psychologically appealing 






C 


1 


It is written clearly 


8 


Too wordy 






It is good for comparison 


8 


It is bad 




1 


Contains all Information 


4 


Difficult to locate 



information 
1 Slows comparisons 

1 Form is crowded 

1 It provides no overview 



ATTACHMENT 5 




DATK: June 23, 1971 

TO: Dr. John F, McManus 

FROM: Howard Wolfe '/^'-'^ 

Sl*IUE(T COMPUTER COSTS ASSOCIATED WITH THE PROPOSED FORMATS FOR THE 
CLASS PERFORMANCE REPORTS 
COPIES TO: J, Hooper, B. Roffman 



The Class Performance Report consists of the percentage of the 
class reaching criterion on each outcome for a past range of 
units* 



The proposed formats are: 



A) Vertical bar graphs for each unit and outcome 

B) Matrix (unit vs. outcome) 

C) Descriptive sentences for each outcome and unit 

See Attachment 2 for a sample of each format* Note that formats B 
and C include the N^s as well as the percentages* 

For each report the number of computer lines printed and the average 
length of each line can be used to estimate printing charges and 
transmission times* 



These figures for a 10 unit SWRL program are supplied in Table 1 
(number of print lines/report) and Table 2 (length of print lines 
in characters)* 



Printing Charges 

Table 1 shows that format B involves the least number of print 
lines per report. Thus this format would be the most economical 
on a system with a flat charge for each page printed. 



Transmission Times 



For remote sites using a TTY as a receiving device the time to 
receive each report is determined by the number of characters in 
the report. The formula to calculate the number of characters per 
report is: 



T=AXB+CXD where 



A := number of heading lines 

B ^ length of each heading line 



^ 16 
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Dr. John F. McManus 
June 23, 1971 
Page 2 

C a number of data lines 

D = length of each data line 

T « nuirf)er of characters/report 

The calculations for each format using the average figures 
over 10 units in Tables 1 and 2 are: 

Format A 

12 X ,42 + 30 X 66 = 2484 characters 
Format B 

12 X 47 + 8 X 51 - 972 characters 
Format C 

15.6 X 60 22 X 64 « 2338 characters 
This again shows B to be the most efficient. 
CPU Time and Computer Memory 

Format B is the most economical with respect to CPU time. Format A 
is the most costly. Computer memory is not significantly different 
for any of the formats. 
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